The present invention relates to a multi-winners feedforward neural network. More to particularly this invention relates to a neural network which is used for a leaning type pattern recognition device such as character recognition, voice recognition, picture recognition and so forth.
All kinds of models are proposed as architecture for realizing neural network which is constructed by neuron-elements with learning function. Such the kinds of models are the perceptron, the recurrent type network, the Hopfield network, the Neo-cognitron, the error back-propagation method, the self-organizing map method, and so forth. The individual function of these models can be achieved by the technique of the analog or digital circuit, further, the individual function of these models can be achieved by the technique of the program in which function is described using the processor as the fundamental circuit. Furthermore, the individual function of these models can be achieved by the both techniques which are combined.
In the neural network system that the analog circuit is taken to be fundamentals, the neuron-element which processes input signal obtains the product-sum between a plurality of input signals and the weights with respect to individual input signal as the state-function of neurons. The product-sum is obtained by using an analog multiplier such as Gilbert amplifier and so forth, or an adder. The output of the product-sum is compared with a threshold value held by respective neuron-elements using a comparator. Then these results are processed in function transformation which is carried out according to the designated function such as Sigmoid function and so forth to output a result of processing of function transformation. Concretely, in xe2x80x9cAnalog VLSI and Neural Systemxe2x80x9d by Carve Mead, 1989, or in xe2x80x9cNeural Network LSIxe2x80x9d by Atsushi Iwata and Yoshihito Amemiya, 1996 published by the Institute of Electronics, Information and Communication Engineers, above described techniques are described.
In the latter system that the processor circuit is fundamentals, all kinds of processing are described by the program. All kinds of processing such as the product-sum function of neurons, the comparison function of neurons, the function-transformation function of neurons and so forth are described by the program, thus being distributed to a plurality of processors to be calculated in parallel.
Furthermore, characteristic functions of a part of neuron-element such as the product-sum function, the pulse output function and so forth are realize by the analog circuit, before carrying out analog digital conversion so that remaining functions are processed in digital method, namely the processing is arranged by the method of compromise between the analog method and the digital method.
When LSI (Large Scale Integration) is constituted by the neuron-element in the conventional system, it is necessary to carry out learning in the neural network with respect to all kinds of subjects in every problem without difficulty. In order to realize flexible adaptation of the neural network as the LSI to the problem, it is desirable that respective neurons are capable of learning the subjects autonomously (referring to unsupervised learning system).
As the unsupervised learning method, Hebbian learning system and self-organizing map method et. al. are known. In the Hebbian learning method that competitive leaning between neurons in the same layer is carried out, some neurons in a group within the neural network are constituted in layered shape. Only one cell of neurons in the respective layers is permitted to be excitatory as the winner. As is clear from the conventional achievements that self-learning is capable according to a single-winner method, hereinafter called as winner-take-all method, due to competition between neurons in the same layer. In the self-organizing map method of Kohonen, a winner-take-all is selected. The learning is carried out regarding only the neurons adjacent to the winner. However, according to the winner-take-all method, it is necessary to provide the neurons for the number of items which are intended to classify to be extracted as the special feature, thus there is the problem that the number of the neurons of the output layer is dissipated.
On the other hand, the error back-propagation method denotes suitable learning efficiency as a supervised learning. The error back-propagation method obtains the different error between an output obtained from the presentation of an input signal and teacher signal expected as an output, before carrying out the learning while changing the weight of the intermediate hidden layer of the front stage, and the weights of the output layer so as to render the error small. The error back-propagation method does not require the winner-take-all method, therefore there is no problem of dissipation in connection with the number of the neurons. However, the error back-propagation method requires teacher signal, and requires complicated calculation for updating the weights at the time of learning, therefore, there is the problem that the circuit of the data processor of the neurons in case of making into the LSI become complicated.
There are disclosed xe2x80x9cA Self-Organizing Neural Network Using Multi-Winners Competitive Procedurexe2x80x9d and xe2x80x9cMulti-Winners Self-Organizing Neural Networkxe2x80x9d by Gjiongtao HUANG and Masafumi HAGIWARA. The former is published in 1995 as the report of THE INSTITUTE OF ELECTRONICS, INFORMATION AND COMMUNICATION ENGINEERS (TECHNICAL PEPORT OF IEICE, NC 94-94 pp 43-150). The later is published in 1997 as the report of THE INSTITUTE OF ELECTRONICS, INFORMATION AND COMMUNICATION ENGINEERS (TECHNICAL PEPORT OF IEICE, NC 96-156 pp 7-14). These are the method for reducing the number of the neurons while permitting multi-layers. The multi-winners self-organizing neural network according to HUANG and HAGIWARA and so forth obtains the sum total of the output of the competitive layer at the time of updating learning of the weights. This neural network prevents dissipation and divergence of the weights while normalizing the output of the competitive layer by using the obtained the sum total. However complicated procedure is required for normalization in that the sum total of the output should be obtained, before normalizing respective output values with obtained the sum total, thus there is the problem that it causes the circuit of the neurons to be complicated at the time of making into the LSI.
In view of the foregoing, it is an object of the present invention, in order to overcome the above-mentioned problems, to provide a multi-winners feedforward neural network in which the neural network is constitute using neuron-element which is obtained while causing the neural network to be made into model, and which can carry out learning autonomously with respect to any problem, and which can be constituted by small number of LSI circuit simply.
According to a first aspect of the present invention, in order to achieve the above-mentioned objects, there is provided a multi-winners feedforward neural network in which a plurality of neurons constituting hierarchical structure of one layer and/or plural layers which neural network has a controller for controlling the number of firing of the neurons, in which the number of firing of the neurons is more than two, in such a way that the number of firing of the neurons is restrained depending on a specified value and/or range of the specified value in every layer.
Consequently, there are more than two pieces of neurons which are excited in respective layers, therefore, it is not necessary to carry out complicated processing of the well known winner-take-all method of one excited neuron. The multi-winner feedforward neural network of two neurons is capable of being realized simply on the ground that a threshold value is set in every respective layers for causing the neuron to be excited.
According to a second aspect of the present invention, in the first aspect, there is provided a multi-winners feedforward neural network, wherein the controller for controlling the number of firing of the neurons, in which the number of firing is more than two, in such a way that the way that the number firing of the neurons is restrained depending on a specified value and/or the range of the specified value in every layer, the controller has a detection controller for causing a value to be the specified value and/or the range of the specified value, depending on obtained value while detecting power supply current supplied to the whole neurons in every layer.
Consequently, power supply current supplied to the neuron of the layer concerned represents the number of firing of the neurons. The value obtained by detecting the current value is function-transformed to be carried out feedback, resulting in control of firing of the neuron of the layer concerned. Thus restriction of the number of firing of the neurons is restricted.
According to a third aspect of the present invention, in the first aspect, there is provided a multi-winners feedforward neural network, wherein the controller for controlling the number of firing of the neurons, in which time number of firing of the neurons is more than two, in such a way that the number of firing of the neuron is restrained depending on a specified value and/or range of the specified value in every layer, the controller comprises a detection controller for controlling the whole neurons of intermediate layer which neurons detect power supply current, causing a value to be the specified value and/or the range of the specified value, depending on obtained value while detecting power supply current supplied to the whole neurons in a plurality of layers of intermediate layer, and a detection controller for controlling the whole neurons of an output layer which neurons detect power supply current, causing a value to be the specified value and/or the range of the specified value, depending on obtained value while detecting power supply current supplied to the whole neurons in the output layer.
Consequently, power supply current supplied to the neuron of the intermediate layer represents the number of firing of the neurons. The current supplied to not only one layer of the intermediate layer but also a plurality of the layers is detected to be function-transformed together, subsequently, the value obtained is carried out feedback, thus controlling firing of the neuron of the layer concerned which detects power supply current of the intermediate layer. For this reason, restriction of the number of firing of the neuron is carried out regarding the neurons of a plurality of the intermediate layers simply.
According to a fourth aspect of the present invention, in the first aspect, there is provided a multi-winners feedforward neural network, wherein the controller for controlling the number of firing of the neurons, in which the number of firing of the neurons is more than two, in such a way that the number of firing of the neurons is restrained depending on a specified value and/or range of the specified value in every layer, the controller comprises a detection controller for controlling the whole neurons of an intermediate layer and an output layer which neurons detect power supply current, causing a value to be the specified value and/or the range of the specified value, depending on obtained value while detecting power supply current supplied to the whole neurons in the intermediate layer and the output layer.
Consequently, power supply current supplied to the neurons of the intermediate layer and the output layer represents the number of firing of the neurons within the intermediate layer and the output layer. The network causes the current value detected to be carried out feedback to control firing of the whole neurons of the intermediate layer and the output layer so that restriction of the number of firing of the neuron is carried out regarding the whole neurons of the intermediate layer and the output layer simply.
According to a fifth aspect of the present invention, in any of the first aspect to the fourth aspect, there is provided a multi-winners feedforward neural network, wherein the controller for controlling the number of firing of the neurons, in which the number of firing of the neurons is more than two, in such a way that the number of firing of the neurons is restrained depending on a specified value and/or range of the specified value in every layer, the controller comprises a detector for detecting power supply current value, a means for function-transforming the power supply current value detected, and a modifier for controlling so as to render the number of firing of the neuron excitatory and/or so as to render the number firing of the neuron inhibitory regarding the neurons of the respective layers depending on the value function-transformed.
Consequently, the detector for detecting current value, the means for function-transformation, and the modifier for controlling firing of the neurons are realized easily using respective well known simple semiconductor circuits.
According to a sixth aspect of the present invention, in the fifth aspect, there is provided a multi-winners feedforward neural network, wherein the modifier for controlling so as to render the number of firing excitatory and/or so as to render the number firing inhibitory regarding the neurons of the respective layers depending on the value of function-transformed, transformed the modifier comprises a means for controlling the number of firing in such a way that when the number of firing of the neurons in the respective layers is less than the required number, the means renders the number of firing of the neurons excitatory, while when the number of firing of the neurons in the respective layers is larger than the required number, the means renders the number of firing of the neurons inhibitory.
Consequently, excitation and inhibition of firing, for instance, is realized simply by combining output of the modifier as one of the weights which constitutes the neurons.
According to a seventh aspect of the present invention, in the fifth aspect, there is provided a multi-winners feedforward neural network, wherein a function of the means for function-transforming the power supply current value detected is a linear transform function and/or a non linear transform function.
Consequently, these linear transformation functions are realized simply by combination of a current mirror, an amplifier and an inversion circuit and so forth, further non linear transformation functions are realized simply by a non linear element which generates an exponential curve and a secondary curve.
According to an eighth aspect of the present invention, in the first aspect, there is provided a multi-winners feedforward neural network, wherein the controller for controlling the number of firing of the neurons, in which the number of firing of the neurons is more than two, in such a way that the number of firing of the neurons is restrained depending on a specified value and/or range of the specified value in every layer, the controller comprises a number of firing detection neuron for detecting the number of firing of the neurons in every layer, thus causing value of output of the number of firing detection neuron to be the specified value and/or the range of the specified value.
Consequently, the number of firing detection neuron receives the output from the whole neurons of the layer concerned, and detecting the number of firing of the neuron within the layer concerned, thus controlling firing of the neuron of the layer concerned.
According to a ninth aspect of the present invention, in the eighth aspect, there is provided a multi-winners feedforward neural network, wherein the number of firing detection neuron consists of a minimum value detection neuron for coping with the case where the number of firing of the neuron decreases to the value less than the minimum value of the specified range in every layer, and a maximum value detection neuron for coping with the case where the number of firing of the neuron exceeds the value larger than the maximum value of the specified range in every layer, in which the minimum value detection neuron renders firing of the neurons of the layer excitatory, and the maximum value detection neuron renders firing of the neurons of the layer inhibitory.
Consequently, the minimum value detection neuron receives the output from respective neurons of the layer concerned as the input, when the total-sum of the input is reached to the minimum value of the number of firing specified, the threshold value is set so as to change the output, resulting in detection of the minimum value. The maximum value detection neuron receives the output from respective neurons of the layer concerned as the input, when the total-sum of the input is reached to the maximum value of the number of firing specified, the threshold value is set so as to change the output, resulting in detection of the maximum value.